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a data Imk (4) that further includes an optical fiber controller (1) coupled to eadi data link (4). Each controUer (1) is interconnect- 
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1 

FIBER OPTIC MEMORY COUPLING SYSTEM 
FXE^P Of THE JtNVfirJTJiPy 

This invention relates to a novel fiber optic nenosry 
interconnection for linking special nenory busses of processing 
nodes and has particular application for real time data processing 
systems operating at large distances. 

BACKGROUKD OF THE INVENTIQW 

Systems for updating memories in coupled nodes are known from 
U.S. Patent No. 4^991,079 the content of which is here incorporated 
by reference and from U.S. Serial No. 07/403,779 filed September 8, 
1989, which is a continuation of Serial No. 06/880,222 filed June 
30, 1986, now abandoned, the content of which is here incorporated 
by reference > all of which are commonly owned with this present 
application. Such systems use two ported memories and are used to 
transfer writes to one memory in one node automatically and at high 
speed to memory in other nodes with the intervention of a CPU. 
Such systems, however, have a distance limitation of about 120 feet 
and eight nodes. The present invention is an improvement that 
enables such systems to be connected over a distance. The present 
state of the art allows for connections of 3 kilometers and up to 
ten kilometers with a high speed data interface. 

SUMMARY OF THE TNVENTTQW 

The present invention provides a means for connecting such 
memory coupled processing systems over a large distance and 
provides for high speed data transfers between the systems, copying 
data fro© the memory of a node in one system to the memory of a 
node in another systeA. 

Other and further advantages of the present invention will 
become readily evident from the following description of a 
preferred embodiment when taken in conjunction with the appended 
drawings. 
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3 Figure x. j-**- *4Kay to mmoxv coupled 

4 Figure 2 illustrates the fiber to memory 

13 Figure 7 illustrated Quadrant 0 of the EMC; 

Figure 8 illustrates Quadrant 1 of the FMC; 
Figure 9 illustrates Quadrant 2 of the FMC; 
Figure lO illustrates Quadrant 3 of the FMC; 
F^^re 11 illustrates a parallel/serial latch used xn the 

18 present invention? 

la Figor. 12 illOTtrKe. the basic packet fonrnti 

,0 13 illuetrate. the tor»t fcr . tota padcet: 

" Fi^e 14 m».tratee the £or«t tor a ,«.«:al purpose ^ 

i: •^'InLrr/mostxatee the ror^t .or ^ i— t 

" M^e 17 muetrate. the ■«<.. =P«."<.» °« «- «<= «" 

r„ ''"'s^i»Tur:t~in,o.™te..- - 

20 lUustrate. a„ «««le of .-t^^X cl«ter. 
n °' "in^ 2X llluetrat.. a .econaary ha<*«P high .p..* linK for 

34 the system; .^^nhack for memory write 

35 Figiire 22 illustrates internal loopback for mem y 

36 transfers; 
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Figure 23 illustrates external loopback for memory write 
transfers; 

Figure 24 illustrates memory coupling bus loopback; 
Figure 25a illustrates internal loopback for general purpose 
asyhc data; 

Figure 25b illustrates external loopback for general purpose 
async data; 

Figure 25c illustrates port-port loopback; 

Figure 26 Illustrates cluster FMC error handling; 

Figure 27 illustrates hub FMC error handling; 

Figure 28 illustrates typical configurations of the present 
invention; 

Figure 29 illustrates a star configuration ot the present 
invention. 

DETAILED DESCRIPtlOlf DF THE P REFERRED EMBODIMENT 

This invention relates to a high speed data interface system - 
using fiber optic memory interconnection as shown in Figure 1. 
This interface system connects memory coupled systems over 
distances up to 10 kilometers. Each memory coupled system 
comprises a data link or memory coupling bus 5 to which up to eight 
nodes 6 are coupled each comprising a processor, I/O, a two or more 
ported memory and a processor to memory bus. Write/read sense 
controllers couple the busses and the memory so writes only are 
sensed and reflected to the memories of other nodes without CPP 
intervention. This is described in detail in the patent and 
application noted above, both of which are here incorporated by 
reference. The bus 5 of each memory coupled system is connected to 
a fiber-to-memory coupling system controller (FMC) l. Each FMC has 
both an input and output port for connection with another FMC. The 
input port 2 is for receiving transmitted data from another memory 
coupled system and the output port 3 is for transmitting data to 
another memory coupled system. The transmission of the data is 
through fiber optic cables 4. 

The Fiber Optic Memory Coupling System (FOMCS) also provides 
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2 the capability for nodes on separate MC busses to exchange async 

3 serial data across the fiber link and for a node on one KC bus to 

4 interrupt a node oh another. Badh FMC supports 8 general purpose 

5 async ports and 16 interrupt lines (B input and 8 output) . the 

6 multidrop console link incorporated in the MCS-il fourth cable is 

7 also supported. These features are intended to allow remote 

8 booting of nodes across the fiber and to provide a means of 

9 synchronizing the actions of nodto. 

10 Thrdughbut this dbciiment^ the term »»MCS cluster" (or simply 

11 "cluster") is used to refer to a MC bus and its attached nodes and 

12 FMC- In configtirations of only two or three clusters, pairs of 

13 VUC8 arfe used to directly connect MC busses (see Figure 28) . In a 

14 configuration of more than three clusters, a FOMCS hub is used to 

15 connect all the clusters in a star configuration (see Figure 29). 

16 The configuration programming link shown in Figures 28 and 29 

17 is used to establish the operational mode of each FMC. FMCs which 

18 are hot directly connected to the prograinming link receive 

19 programming information via packets sent over the fiber link. The 

20 FMC 1 can be seen in more detail in Figure 2. The FMC includes a 

21 Receive Data Pith 7, an output latch 9, a receive FIFO ii, a 

22 receive Error Detection Circuit 13, two receive latches 15 and 17, 

23 a receiver 19, an input latch 10,. a hit and translation RAM 12, a - 

24 transmit FIFO 14, a transmit Error Detection Circuit 16, two 

25 transmit latches 18 and 20 and a transmitter 22. 

26 The FMC 1 consists of four main sections: the memory coupled 

27 system 5 Interface, the data paths (Rx and Tx) 7 and 8, the high 

28 speed serial data link interface 4, and the microprocessor • These 

29 coreas; are delineated uding dashed lines in the functional block 

30 diagram (Figure 2) . The Fiber Transition Module (FTM) connects to 

31 the EMC high speed serial link interface. The FTM is a known and 

32 conventional piece of hardware and serves to connect the electrical 

33 signals to corresponding light signals and vice versa. 

34 The Memory Coupling (MC) bus interface is the FMCs link to 

35 the Memory Coupling System (MCS) and the nodes on that bus. The 

36 FMC implements a three-cable, 90-signal interface for those MCS 



y 
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2 networks that use a standard MC bus (24-bit addresses), and a 

3 four-cable, 120-signal interface for those MCS networks that use 

4 the 28-bit addresses provided in HCS-II. Provision is also made in 

5 the four-cable interface for eventual support of 32-bit addresses. 

6 The PMC supports all the address lines defined in the four-cable 

7 interface; however, in an environment where addresses are a full 32 

8 bits, the FMC will only reflect into and out of the first 256 

9 neigabytes of memory. 

10 The FMC appears as a typical MCS node on the bus, using one 

11 of the nine bus IDs (0 - $) available in MCS or MCS-II, The MCS 

12 used by the FMC is set during initialization by the configuration 
a' 13 . , programming link* 

14 Details 6f the FMC interface to the MC bus are illustrated in 

r t ) 15 Figure 3* The FMC MC bus interface receives bus transfers by 

V 1^ latching the memory address, memory data, flag bits and parity on 

17 the rising edge of DATA VALID (assuming DATA VALID is not being 

18 driven by the FMC itself). The two different, bit counts for the 

19 address, flags and parity lines reflect differences between the 

20 three cable HCB bus and the four cable MCS-IX biis* The smaller 

21 counts apply to the three cable bus. Kote that 32 bits of address 

22 are indicated for the four cable MCS-II bus rather than 28. As 

23 mentioned earlier, signals to. support the additional four address 

24 bitis are reserved in the fourth cable for future expansion. One 

25 bit of odd parity is provided for each byte of address and data, 

26 resulting in the seven or eight parity bits shown in Figure 3. 

27 The flag bits cjualify the received transfer. For memory 

28 write transfers, one flag bit (referred to as the "F-bit«) 

29 indicates whether the memory write is to a byte location in memory 
-^30 as opposed to a halfwbrd or word. The other two flag bits, which 

31 are only present in the MCS-II bus, differentiate between memory 

^ 32 write transfers and other types of transfers. While only memory 

33 write transfers appear on an MCS-II bus in a cluster, the MCS-II 

34 bus in the FOMCS hub is used to distribute interrupt, async and 

35 other types of data in addition to memory write traffic, 

f 36 As indicated in Figure 3, the address, flags and parity are 
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2 treated as 32, 3 and 8 bit quantities, respectively, when received 

3 by the FMC. If a three cable MCS bus is connected to the FMC, the 

4 receivers for the eight most significant address bits, the tvo 

5 e3Ctra flag bits and the eighth parity bit are disabled. Zeros are 

6 put in the MC input latch for the missing addreiss and flag bits and 

7 a one is inserted for the missing parity bit. In a fbtxr cable 

8 environment, 32 bit addresses are received but the FHC will discard 

9 any received memory write transfer in which the most significant 

10 four address bits are not zero. 

11 If receipt of a transfer causes the FMC Tx FIFO 14 to become 

12 half full, the FHC will drive GLOBAL BUSY on the MC bus to prevent 

13 overflow. GLOBAL BUSY is deasserted when the Tx FIFO 14 becomes 

14 less than half full. The CONTROL BUSY signal is only present in f 

15 the KCS*iX bus and is only utilized in the FOHCS hub. If receipt 

16 of a non'-memory-write type transfer causes the FMC FIFO used to 

17 hold such trsmsfers tb become half full, the FMC will drive CONTROL 

18 BUSY to prevent overflow. When the FIFO becomes less than half 

19 full, CONTROL BUSY is deasserted. 

20 Before the FMC can generate a tremsfer on the MC bus, it must 

21 first acquire the right to access the bus. To accomplish this, the 

22 FKC asserts the REQUEST line on the bus which corresponds to the 

23 HCS ID the FMC has been programmed to use. The FMti then monitors 

24 the corresponding GRANT line. When the bus arbiter asserts the 

25 GRANT line, the FMC drives the memory address, memory data, flag 

26 bits and parity on the bus. The FMC's MCS ID is also driven on the 

27 node id lines of the bus. On the nert rising edge of the VALID 

28 l^ABLE signal, the FMC drives DATA VALID. (The VALID ENABLE signal 

29 is a free running clock generated by the bus arbiter to synchronize 

30 the actions of nodes on the bus.) ^ 
-31 Note that the MC output latch illustrated in Figure 3 only 

32 supplies 28 bits of the memory address rather than 32. The driver 

33 inputs for the remaining four address bits are tied to ground 

34 (i*e., the bits are forced to be zeros). 

35 If receipt of a memory write transfer packet causes the FMC 

36 Rx FIFO 11 to become half full and MC bus burst request mode is ^ 
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enabled, the FMC will drive BURST REQUEST. Asserting the BURST 
REQUEST signal r which is only present in the ttCS-li bus, causes the 
bus arbiter to enter a mode in which only the FMC is granted access 
to the bus. After asserting BURST REQUEST and delaying long 
enough to guarantee propagation pf the signal to the arbiter, the 
FHC deasserts GLOBAL hvs2. If no other node on the bus is 
asserting GLOBAL SUSY, the arbiter will begin issuing grants to the 
FMC. If orie or aore other nodes is asserting GLOBAL BUSY^ the 
arbiter waits for the busy condition to clear and then begins 
issuing grants to the FMC, Note that the FMC can safely deagsert 
GLOBAL BUSY even if it is vmable to accept more transfers from the 
bus because the arbiter will only grant bus access to the FMC. 

When the FMC has unloaded enough packets front its Rx FIFO 11 
to cause the fill level to drop below half full, BURST REQUEST is 
deasserted and the arbiter returns to the normal «fairness« 
arbitration scheme. If the FMC is unable to accept more transfers 
from the bus, it will assert GLOBAL BUSY before deasserting BURST 
R£QU]^ST. 

If the FMC's Tx FIFO is less than half full, the FMC will 
keep BURST REQUEST asserted for 8 MC bus cycles and then release it 
for 16 cycles and then assert it for 8, release it for 16 and so on 
until the Rx FIFO fill level drops below half full. If both the 
RX and TX FIFOs are half or more full, the FMC will assert BURST 
REQUEST continuously until enough packets have been iinloaded from 
the Rx FIFO to cause the fill level to drop below half full. Once 
BURST REQUEST is deasserted, the arbiter returns to the normal 
"fairhisss" arbitration scheme. If the PMC is unable to accept more 
transfers from the bus, it will assert GLOBAL BUSY before 
deasserting BURST REQUEST. 

support for burst request mode in the FMC is enabled or 
disabled via a configuration programming command. Only one FMC on 
a given MC bus can have burst request mode enabled but at least one 
FMC in every FHC-*to-FMC link must have the mode enabled to enst^ re 
reliable operation. Note that enabling burst request mode in ti4e 
FMC only means that the FMC is . able to drive BURST REQUEST on the 
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bus. For burst request mode to be useful, tlie four cable MCS-II 
environment with Memory Coupling Controllers tMCC's) as the bus 
arbiter/teradnators is required. MCC's provide bus termination and 
arbitration in a conventional manner as known from the previously 
noted patent and application and are sometimes referred to as 
reflective memory controllers. 

The FMC Rx aiid Tx data paths 7 and 8 move data between the 
MCS interface and the high speed serial data link. *hey also 
interface to the microprocessor allowing asynchronous data, 
interrupt transfers and flow control information to move between 
clusters. 

Figure 4 illustrates the maxuier in irtiich the EKC processes 
MCS ifiembry write transfers. As indicated r not all transfers are 
transmitted as packets over the high speed serial data link. Only 
thbse transfers which correspond to memory writes into selected 
regions of memory are transmitted over the high ispeed link. 
Transfers whicdi do not fall within these selected regions are 

simply discarded. 

In the other direction, packets containing memory write 
transfers are received over the high speed link and decoded by the 
FMC. Note that in the Rx path 7 there is no hit/translation RAM so 
all memory write packets received over the high speed link cause 
memory write transfers to be generated on the MC bus. When a 
packet has been received which represents a write into memory, the 
FMC requests the use of the MC bus and when the request is granted, 
generates a memory write transfer on the bus* 

In the tac path 8, the regions of memory to be reflected are 
defined during configuration prograiaming of the FHC. During 
programming, the total memory address space is viewed as a sequence 
of 8K byte {2K word) blocks. Multiple regions as small as 8K bytes 
can be defined. Any number of regions, segregated or concatenated, 
up through the entire address range may be established. 

toother feature of the FMC which is illustrated in Figure 4 
is memory address translation. Prior to packetizing a memory 
write transfer and transmitting it over the high speed link, the 
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memory address is modified, MCS physical addresses are 24 bits 
long (MCS-I) or 28 bits long (MCS-II) . A MCS physical address is 
translated by using the most significant li bits (HCS-I) or 15 bits 
(MCS-II) to address a hit/translation RAM on the FMC. The value 
read from the RAM is the most significant 15 bits of the new 
address. 

Note that the least significant 13 bits of a memory address 
are unaffected by the translation process « ' ihns, 8)c byte blocJcs 
are mapped from the address space of the source MC bus to that of 
the destination MC bus and vice verse. As illustrated in Figure 5, 
this feature is very useful because it allows clusters to share 
memory regions which reside in the different places in each 
cluster^ s physical address space. 

The contents of the hit/translation lUUf are established 
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1 during configuration programming of the FRC. The size of memory 

2 addresses (i.e., 24 or 28 bits) on the source MC bus is also 

3 established duting configuration programilng. Each location in the 

4 hit/translation RAM represents a 8K byte blo<flc of memory and 

5 contains a hit bit and the 15 bit translation value. If the hit 

6 bit is set, memory writes into the 8K byte memory block are 

7 reflected over the high speed link. If the bit is reset, such 

8 memory writes are ignored. 

$ m the content of the MCS star network configuration, the FKC 
region seleistion and address translation features can be used to 
segregate the MCS clusters connected to the hub into groups such 
that memory write traffic is only reflected within a grottp, not 

13 between groups. Hote that if an administrative system (i.e., not 

14 one of the nodes in the star network) is used to program the FMCs, 

15 a secure network can be achieved in which cluster groups can 

16 coexist but not affect each others mwttory. If total isolation of 

17 groups is not desired, overlapping regions can be used. 

18 The FMC microprocessor has the ability to interject data into 

19 - and remove data froa both the Tx and Rx data paths 7 and 8. such 

20 data is referred to as control data to distinguish it from memory 

21 write transfers. General purpose async data, MCS-Il multidrop 

22 console async data and interrupt pulses are all treated as control 

23 data. 

24 When the FMC receives data over the general purpose async 

25 ports, it fortts the data into packets and injects those packets 

26 into the transmit data stream sent over the high speed link. When 

27 a packet of async data is received by the PMC, the bytes are broken 

28 out of the packet and sent over the appropriate general purpose 

29 async port. MCS-II multidrop console data is handled in a similar 

30 fashion. 

31 When an enabled input interrupt line is pulsed, the IMC 

32 generates a packet that is sent over the high speed link, upon 

33 receipt of an interrupt packet from the serial data link, the BMC 

34 pulses the appropriate output interrupt line. 

35 other control transfers include configuration programming 
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1 informatioh, high availability messages/ error indications « and 

2 reset indications, 

3 Flow cbnteol in a MCS netvork is accomplished by means of 

4 f V control bits in the packets sent from FMC to FMC and by means 

5 o£ MC bus busy signals, internal to the FMC, memory write data 

6 transfers and other types of data transfers (i.e., async, 

7 interrupt, multidrop, etc) are handled separately so that How 

8 control may be applied to non-mjemory write transftos without 

9 affecting the memory write traffic. Each packet sent from FMC to 

10 FMC contains two flow control bits, one to cause the receiving FMC 

11 to cease or resume transmission of memory write transfer packets 

12 and another to cause it to cease or resime transmission of other 

13 typies of packets. This notion of two separate data streams also 

14 -- applies to the hub MC bus where there are separate bus busy 

15 signals, one for memory wrlt« transfers and another for all other 

16 types of transfers. 

17 Ah additional type of flow control is burst request mode. 

18 Burst request mode is n«»ces8ary to ensure that lock ups do not 

19 occur on FMC-to-^niC links. When link utilization is high in both 

20 directions, the potential exists for a FMC- to-FMC link to lock up 

21 in a condition in which both FMCs are asserting busy on their 

22 respective MC blisses. Because the busy condition on the MC busses 

23 prevents the FMCIb from generating bus transfers, the FMCs are 

24 unable to reduce the fill levels of their ^ FIFOs 14 and will 

25 therefore never deassert bus busy. 

26 Burst request mode alleviates this problem by allowing a FMC 

27 to unload transfers from its Rx FlFO 11 while ensuring that the EMC 

28 will not have to accept additional transfers into its Tx Flf^ 14. 

29 This means the FMC can accept more memory write trainsfer packets 

30 from the remote FMC which in turn allows the remote FMC to unload 
' 31 its Tx FIFO 14 and eventually clear the busy condition on the 

32 remote MCS bus. Clearing the busy condition allows the remote FMC 

33 to unload its Rx FIFO 11. The remote File can then accept more 

34 memory write transfer packets which allows the local FMC to unload 

35 its Tx FIFO 14 end clear the busy condition on the local MC bus. 
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A. shown in Figure 6. the FHC data paths are logically 
subdivided into quadrants. In quadrant 0, MC bus transfers are 
received and noved to the Tx FIFO 14. Thft Tx FIFO 14 serves as the 

4 boundary between quadrants 0 and 2. In quadrant 2. transfers 

5 renovM from the Tx FIFO 14 are paoKetized and transmitted over the 

6 high speed serial link 4. 

7 Packets are received over the high speed link 4 in quadrant 

8 3 and the packet contents are moved to the Rx FIFO 11. the Rx 

9 FIFO 11 serves as the boundary between quadrants 3 and i. In 
lb quadrant 1, information removed from the Rx FIFO 11 is used to 

11 generate transfers on the MC bus. 

12 An important feature of the FMC design, particularly from a 

13 diagnostic perspective, is that the latcShes ih each quadrant can 

14 be accessed in both a parallel and serial fashion. During normal 

15 operation of the FMC, data moves through the latches using the 

16 parallel interface. The alternate serial interface allows the 

17 microprocessor on the IMC to shift data serially into and out of 

18 the latdhes. 

19 Figure 7 shows a detailed view of quadrant 0. Processing 
2d begins in quadrant 0 when the control logic 25 detects that the MC 

21 input latch 10 has been loaded. The input latch is unloaded and 

22 odd parity is computed on the address and data. The cdmputed 

23 parity is then compared to the received parity. At the same time, 

24 the most significsnt address four bit. are check«i to see if any 

25 of them are non-zero. 

26 While the parity and address checks are taking place, 15 bits 

27 of the address are used to address the hit/translation RAit 12. (If 

28 the address came from a three cable MCS bus, the upper four of th» 

29 15 bits used to address the RAM will be zeros.) The least 

30 significant 15 bits of the 16 bit value read from the RAM hecome 

31 the new, or translated, address bits. The most significant bit 

32 of the value read from RAM is the window hit bit. 

33 The destination of the MC transfer is determined by the 

34 parity and addrsss iShecks, the hit bit, and the flag bits received 

35 With the transfer. If any of the most significant four address bits 
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1 is non-zero, the transfer is discarded. If the received parity 

2 does not equal the computed parity, the transfer is clocked into 

3 the microprocessor interface FIFO 2X* The transfer is also placed 

4 into the micro FIFO if the parity is gockl but the flag bits 

5 indicate a control type transfer (which only occurs in the MCS 
- 6 hub) • The destination of a memory write type transfer with good 

7 parity, depends on the setting of the hit bit. If the hit bit is 

8 reset, the transfer is simply discarded. If the hit bit is. set, 

9 the m^ory data, translated memory address and flag bits are 
16 clocked into the Tx FIFO 14. 

XX The contents of the hit/translation 12 are initialized 

12 by t^e FHC microprocessor. To change a location in the RAM 12, the 

13 microprocessor first puts the new value into the loading .buffers 
14 24. The micrppirbcessor then causes GLOBAL BUSV to be asserted On 
is the MC bus and waits long enough for any transfer in progress to 

16 pass through the MC input latch 10. Then, the microprocessor 

17 causes the hit/translation lUUi 12 to be written with the value from 

18 the loading buffers. GLOBAL BtlSY is. subsequently deasserted. 

19 Figure 8 shows a detailed view of quadrant 1. Quadrant 1 

20 processing begins when the control logic 25 detects that the MC 

21 output latch 9 is empty and either the Rx FIFO 11 is not empty or 

22 the micro interface 21 is requesting use of the latch 9. If there 
iz is something in the FIFO II and the micro interface 21 is not 

24 re^esiting, the FIFO 11 is read and parity is computed on the 

25 meiQory address and data. The data, address, flagis and computed 

26 parity are then clocked into the HC output latch. 

27 The micro interface 21 is only used when a FMC in a FOMC& hub 

28 needs to distribute interrupt, async of other control information 

29 to the other FMCs in the hub. If control busy is not asserted on 

30 the hub bus, the micro interface logic 21 requests use of the MC 

31 output latch 9» When the latch 9 is available, the control data 

32 is serially shifted into the latch 9. Odd parity for the transfer 

33 is computed by the microprocessor and shifted into the latch 9 

34 following the data* 

35 Figure 9 shows a detailed view of quadrant 2. Processing 



wo 93/19422 



PCr/US93/02Je9 



. H«»nt 2 When the control logic detects that the Tx 
, .egins ^J^^^^J^;^ ^ that the Tx riFO 14 U not 

' ''^T'r tl^^e^cropJ^^^^^ has loaded the «icro interface 

3 e»pty been loaded, the contents of the 

4 latch. " ^ letches 18 and 20. Of the 72 bits 

5 latax are ^'^^^^Tl^, 64 a^e also clocK^ i«to . pair 

, t,^^err^J^«--^r^e) gehera^^ ^ 
, Of EDC Cerror de^^ ^. 

' '"^Trmn^^t^ » The contents of the TX 

li:^ rar:o^r: then passed to the high speed linK serial 

- --ii^:sroTr™r«.e..n-uishot,th. 



12 
13 



. ...«t- " " « ci<»l= into th. l.t<*», 0» 

„ <=»'"^'°'""^°^^^. ,ortr.™-i..i.ntot».r«.ot.™c. 
» iirto th. « lrtol« » "« " ' th. K FIFO U ««1 tto 

» «p«- f^^r^nr rt ^ -o: o, 

« . U «.t to th. r»ot. 

27 become less than haxr zaxx, " or both types of 

28 IHC that it can resume transmission of one or both typ 

29 packets. ii-<4 view of ouadrant 3. Quadrant 3 

Figure 10 shows a^-taUed view of ^ ^^^^^^ 

,X processing begins when the first *0j^^/ ,0 

32 over the high J-^^^^^^^, .tti is. When the rest of the 

33 bits to ^^^•^^^^^^^.'J^Tblt pacK^ is clocked int. the Bx 
packet arrives, the ^^^^ ^he packet are moved 
latch 17. From the Kc latch 17, 64 bits oi 



34 

35 latch 17. 
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to a pair of EDC geinerators/checkdrs 13a and 13b. The received EDO 
is compared to that generated and an indication of the outcome is 
provided to the control logic « 

The destination of the contents of the packet depend on the 
packet type bits examined by the control logic and the result of 
the Ebc check. If the EOC check fails # the packet contents are 
moved to the micro interface FIFO 23. the packet conlients are also 
moved to the inicro FIFO 23 if the packet type flags indicate that 
the packet contains async, interrupt or other control information. 
Othervise, the memory address, data and flags are moved ^o the Rx 
FIFO 11. 

Among the 12 packet bits examined by the control logic 25 are 
the flow control bits, one for throttling memory write packet 
transmission and the other for throttling non memory write packet 
transmission. A set flow control bi^ in a received packet causes 
the FMC to cease transmission of the corresponding type of packet 
until a packet is received with that flow control bit reset. 

Figxire 11 shows a block diagram of the AMD 29818 latch used 
to implement the MC input and output latches 9 and 10 and the Rx 
latches 15 and 17 and Tx latches 18 and 20 on the FMC. In the FMC 
design, the normal data path through these latches is the parallel 
one from the D input through the pipeline reg^ister 29 to the Q' 
output. The alternate serial path is accessible by the FMC 
microprocessor and is primarily used for diagnostic purposes. Note 
that the mux 30 (which is controlled by the MODE signal) allows 
either the contents of th^ shadow register 31 or the D input to 
serve as the input, for the pipeline register 29. Also, note that 
the output of the pipeline register 29 is feed back into the shadow 
register 31. Thus, with appropriate hardware control, data can 
enter the latch in serial and exit in parallel or enter parallel 
and exit in sisrial. 

These features are exploited by the FMC design which allows 
the microprocessor to serially load a latch in one quadrant, move 
the data along the normal parallel path from that latch to a latch 
in another quadrant and then serially read the contents of the 
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. i.. i-^^ch Bv eomparing thd data loaded into the first 
I rtT^^t^rt:- Ic. ..con.. «» .i^oproc.... C 

* *-^-:L^Kr.rJtr"ri* ------ 

*»»leBent*i %rith a transit/receive pair of Gazelle Hot Rod chips. 
Z T^X 1 Hod trahs»itt.r c^ip converts 40-hit parallel da^a 
!L "etial dita that is transmitted over a fiber or coax link. 
ITt^ r^^te .nd of the linK. the d.ta is —^^^^ 
Original parall^a data by a Hot Rod receiver chip. Over J*is 1^, 
«C Lhange information in the form of so-hit paclcets. The 
Z^T^^tt^ .ends eaeh SO-bit pacXet as two 40-bit data 
Gazeue «ra™«* . ^ ^ vKof^nfimltter sends 



4 
5 
6 
7 
8 
9 

^^iJT^ ne«l lor « ~l»r.t. clock .U«l to B. mo-fl 
^ "t" " .. X <o« .UUO. Mf p« 
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17 

18 with the data,. 

19 
20 
21 
22 
23 
24 



^^"'iTsmll^nrtLr detection is ac^«pUshed via an ei.ht 
bit error detection code (EDC) which is included in each packet. 
O thT^2 lainin, bits in the packet, only .4 «e ^^^^^ 
EDC calculation. The eight unprotected bits are the first four 
bits of each of the 40-bit halves of the packet. 
25 Bie basic padcet format is shown in Figure 12. 

Bits 0 and 40 are used to distinguish the two 40-bit 
sectiO^ Of the packet. When receiving a packet, e WC always 
^^^tL f ir^Ait Of the first 40^bit section to be . s*ro and 
^ first bit of the second 40-bit section to be a one. 
the fxrst »« or respectively) are used 

The BSD end BSC bits (bxc» ■»■ ^« 

PMC to throttle transmissions by the IMC 
by the transmitting FMC ^ receiving 

receiving the packet. ^1^^^^':,"^^ data packets, 

niC to cease .r "J"^ ^^^^^^J^jf^i,^ to cease or resume 

34 the BSC bit is used to tell the rece g 

35 transmission of control packets. The bits are set 



26 
27 
28 
29 
30 
31 
32 
33 
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receiving FffC to cease the associated type of transmissions; they 
are r'eset to tell t^e receiving FMC to resume transmissions. 

The TYP bit (bit 4) indicates the packet type. if the bit 
is reset, the packet contains a memory write transfer. If the bit 
is set, the packet is designated as a control packet. Control 
packets are used to transfer async data, interrupt data and any 
other type of information which FMCs must exchange. 

The VAL bit (bit 5) indicates whether the cross-hatched 
portions of the packet (bits 6-39 and 44 - 71) actually contain 
any valid information. if the VAL bit is set, the receiving FMC 
will consider the information in the cross-hatched portions to be 
valid. If the VAL bit is reset, the receiving FMC will only pay 
attention to the flow control information encoded in the packet 
(i.e., the settings of the BSD and BSC bits). When a FMC does not 
have any data to sehd, it will periodically transmit packets with 
the VAL bit rieset and the. flow control bits set or reset as 
appropriate. 

As mentioned earlier, bits 0-3 and bits 40 - 43 are not 
included in the £DC calculation. This means that the BSD bit 
and/or the BiSC bit may be in error in a received packet and no EDO 
error will be detected. However, even if an invalid flow control 
indication is received and acted upon, the next packet received 
will almost certainly correct the problem. ^ 

The shaded portions of the packet (bits 2-3 and bits 42 ^ 
43) are reserved. The convention of shading reserved portions of 
packets is followed throughout this documents 

Bits 72 - 79 contain the EDO which is computed on bits 4 - 39 
and bits 44 - 71. The EDO is an 8-bit check byte generated 
according to a modified Heuoming Code which will allow detection 
of all single- and double-bit errors and some triple-bit errors. 
Because of the NRZI encoding scheme used by the Gazelle Hot Rod 
chips, noise on the high speed serial link medium will cause 
sequential double-bit errors which the receiving FMC will be able 
to detect by comparing the EDO in the packet to that computed 
during receipt of the packet. 
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t^',^ r~^v« ««. th. »= bu.. Tt. «<»«t of th. I. 

' r "i« tT. .ill,!, b^ •»« FH>tt U r«« it th. 

^.^,n<f«r represent* a write to a half-word or word. 

K^e^^t if th, »e»ory address is a 24.hit address, bits 

Ojie general purpose async data pacjcet larma 
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transfer packet is used by the FMC to 



8 

9 
16 
11 
12 

" '""^'s**** c« =«.fin .P to fiv. Wt.. « d.ta^ Th. 

14 *'* . . 4»he bVl 

15 



«lr.t^byt« PX>=.« i» tb. £i«t Wt. i^ 

" ^ tL ' -n. ». r.c.ivi„, rKc a.t«*in« th. .=«.i ' 



" t - 't.'isni=r^ri^^:'rr^t:;; 

" ^^int.«.ti««i«>ol».«--.«-~.ti»«»" 
" '*°**!^. i» bit (bit «) l»lio.t« that th. ori,i».tih9 FHC 
58 oaclcet does not contain any asyno data (i.a., tne eyr. 

32 The HCS-II »altidrop eoMol. a«t« pMMt lormi 

" cohteih up te «.«r byte, of "te. Jh. 
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1 8 ^ 15, th0 second in 16 - 23 and so on. The receiving FMC 

2 detennines the actual nunber of data bytes in the packet by 

3 checKing the Byte Count field (bits 45 - 47} . 

4 The Destination Clusteir field (bits 48 - 55) contains the 

5 address of the HCS cluster for which the packet is ultimately 

6 intended. The contentit of this field are only meaningful If the 

7 BRD bit is reset. 

8 The Source Cluster field (bits 56-63) contains the address 

9 of the cluster in vhich the originating FMC resides. Because 

10 multiple paclcets are required to send a multidrop message, a FHC 

11 in a cluster may be receiving pai^tf of messages from two or more 

12 clusters at the same time. The source cluster field in the packets 

13 allows the receiving FHC to segregate message pieces based on 

14 source cluster and thus properly rieconstruct tfie messages. 

15 The SDH bit (bit 64) indicates whether the async data in the 

16 packet represents the start of a multidrop message or not. if the 

17 SOM bit is set, the packet contains the first 1-4 bytes 
lift (depending oh the Byte Count) of a multidrop message. If the SDK 

19 bit is reset, the paeket contains data from within the body of a 

20 . messagie. 

21 The EON bit (bit 65) indicates whether the asyhc data in the 

22 packet represents the end of a multidrop message or not. If the 

23 EOM bit is set, the packet contains the last l » 4 bytes (depending 

24 oh the Byte Count) of a multidrop message. If the £0M bit is 

25 reset, at least one more packet of data from the message will 

26 follow, 

27 'The BRD bit (bit 67) id set if the packet is to be broadcast 

28 throughout the FOMCS network to all clusters. If the BRD bit Is 

29 set, all FMCs in a FOHCS hub will forward the packet to their 

30 respective remote clusters. All cluster FMCs receiving such a 

31 packet will accept it (assuming they have multidrop support 

32 enabled) . 

33 The interrupt packet format is shown in Figure 16. 

34 The Destination Cluster field <bits 48 - 55) contains the 

35 address of the NCS cluster for which the packet id ultimately 
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intended. The FKC In the destlnetion cluster uses *^^;>-«*^"J*^^^^ 
iSrWeld (bits 56 . 63) to determine which output interrupt line 

""T Motorola 68000 aicroproce.«.r on the FMC provides the 
5 configuration syst«i ihterfeo. e. well as tota «anage«e»t for all 
S 6th J than .«aory write transfers. This includes all 
i ^hronous data, interrupts and error logging. It 
I ^stlo capabilities driven through the configuration 

Pro^^^^^^ ^ ^ 3upport for 

11 
12 

14 d.ta is actually traveling over a hl^-epeed fiber 
^* L_ ^^.rti^e of the entiUes,. comnunication is not 

16 
17 
18 

id 

20 



the exchahSes of aeync serial data between the nodes end/or devices 
in separate hcS clusters. The async support is designed^in such 
a way that communicating entities need not be aware that the^ync 
data is actually traveling over a hi^-epeed fiber 
ftre- the persp^StiVe of the entiUes,. communication is not 
functionell j dif f^nt from the case where the nodes «^/- ^evxces 
are directly connected via cables. Ho special protocol 

inf oriaation has to be inserted by the communicating entities into 
Se async data etr.«. to allow the data to move 
networi: Figure 17 illustrate, the handling of async serial data 
11 by a EMC in a cluster. 

22 AS async serial data is received from a node or device the 

FHC oackeU^es the data «id transmits the padtets over the high 
™Lr^^ lit*, m -^i^i- to a*y«c data, ea* 
ZZLss ^hicsh f eciUtates routing of the paclcet through the MCS 
in the other direction, the FMC receives and decodes 
;:::rofrs.ncd.ta from the high speed lin.. ----f^- 
packet is discarded and the async data is passed to the node or 

30 """'"L address in an async p«*et is designed to allow routing 
i Of thHaLt through a MCS hub. - accomplish this, tne address 

31 ot « F destination cluster field and a 

pert field indicates which async link of the fhc in «x 



23 
24 
25 
26 
27 
28 
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34 
35 
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1 cluster that the data is Intended for. The destination EKC 

2 validates the address When it receives the packet. If the 

3 destination cluster address does not match the local cluster 

4 address f the packet is discarded. 

% The FKC naintains a list of eight async addresses, one for 

6 each general purpose asyhe port. When async data is received from 

7 one of the eight ports, the FMC looks up the async address for that 

8 port and includes it in the packet(s} sent over the high speed 

9 link. Thus, there is a static connection between each general 
16 purpose async port of a FMC and an async port of a FMC in some 

11 other cluster « 

12 The contents of the liist of ac^ync addresses and the local 
( y 13 cluster address are established during configuration programming 

^ 14 of the FMC* The physical characteristics of the async links (i.e., 

\i baud rate, character lengthy etc.) are also established during 

16 configuration progrsumning. 

17 In a MCS hub, the FMCs are programmed to operate in a special 

18 async data pass-thrbugh mode. This mode of operation is 

19 illustrated in Figure 18. Async packets received over the high 

20 speed link are decoded by the receiving FMC and sent over the hub 

21 HC bus to the other FMCs in the hub. The address field inserted 

22 by the FMC at the originating cluster is passed along over the bub 

23 MC bus as veil as the async data. Each FMC in the hiab which 

24 receives the address and async data from the MC bus compares the 

25 destination cluster field to the address of the cluster at the 

26 remote end of its high speed link. If a match occurs, the FMC 

27 builds and transmits an async packet over the high speed data link. 

28 The remote cluster address used by a FMC in the hub to perform 

29 the routing function just mentioned is established during 

30 configuration programming, since the async links of a FMC in a hub 
' 31 are not connected and are ignored by the FMC, no async address list 

32 or async link physical* characteristics can be programmed. 

33 To accommodate the case where the high speed link of a FMC in 
^ 34 one hub is directly connected to a FMC in another hub (rather than 

> 35 to a MCS cluster) , async address checking may also be disabled in 
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a niC vi. configuration progra»ming. Thus, all asyn* paclcet 
Lame o^er t.. ^of one .u. wUl also appeat on the bus of the 

r: T^osrof-LTiinT^sTioi^u::. 

asserted. X loss ^Tsa ^ connected 
eaaae tramteission untU CIS reappears. 

port c» throttl. T»C a.t. tr«»l»lo. for ttot 

>.v asserting and deasserting CTS. . ^ ^ 

I^lHlU it d««.«t. MR to i»»r. tte «rtc. c<«m«t«. to th. 

»^ to iMict. that — '^^ ■-;ft;o. 
Dorinj cnlHuratlon pr.,r«»iBg of th. FK, a««o «^o 
^rt^usL OTR CIS «« b. «»bl- « «■ a p« port 

no. oo.*r.l i. «t «*ir«l, «c ,«»r.l purp.« 

" •^rx"ni*:itTrt'«>ich.«ci.o«».ot-toa. 

a^o^tTjL «a=h i. tr«-^tti», data -«h th«. th^ 

It asvnc packets for other async ports is unaffected. 

flow of async pawwi* ^ ilnas* eiaht input lines and 

•MMk FMC supports 16 interrupt lines, exgnx. *«f 
.l^tttj^ l3 «- lih- alio. » intarrupt pul.. ^.ratad 
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1 by a node or device in one MCS cluster to be passed, in effect, 

2 across the MCS netvorK to a node or device in another cluster. The 

3 pulse enters the Flic In the originating cluster via an input 

4 interrupt line, is passed across the MCS network as a packet and 

5 leaves the FMC in the destination cluster via an output interrupt 

6 . line. As With asyne data, this is designed- in such a vay that the 

7 node need not be aware that the interrupt pulse is travelling over 

8 a high speed serial link and not actually directly connected. 

9 An overview of interrupt passing for a FOMCS configuration JLs 

10 shoWh in Figure 19. The process involved in passing an interrupt 

11 across FOMCS is very similar to that used to pass async data. The 

12 FMC detects the pulse on the input interrupt line and constructs 
j 13 a special ihterriipt packet which is transmitted over the high speed 

^ 14 link. The interrupt packet contains an address which facilitates 

15 routing through the fonts network. At the destination tHC, the 

16 packet is decoded and the address is used to determine which 

17 interrupt line should be pulsed • 

18 As in the async case, the address in an interrupt packet 

19 consists of two fields* The destination cluster field identifies 

20 the cluster to i^ich the packet is to be routed. The destination 

21 line field indicates which output interrupt line is to be pulsed. 

22 As vith async packets, the destination FMC validates the cluster 

23 field of a received interrupt packet and discards the packet if the 

24 destination cluster does not match the local cluster address • 

25 The FMC mttintains a list of eight intenupt ^ine addresses, 

26 one for 6ach input interrupt line, linien the FMC detects a pulse 

27 on one of the input interrupt lines, it looks up the line address 

28 and includes it in the packet sent over the high speed link^ Thus, 

* 29 there is a static connection between each input interrupt line of 
30 a FMC and an output line of a FMC in some other cluster » The 

• 31 contents of the list of interrupt line addresses are established 

32 during configuration programming of the FMC, 

33 In a FOMCS hub, the FMCs are programmed to operate in 
( 34 pass-through mode very similar to that described earlier for async 

- 35 data« Interrupt packets received over high speed link are 
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Eaeh FHC 



MC bus to the other FMCs in the hub. Eaeh FHC 
disttibuted over the MC ^sto^^ ^^^^ ^. ^ 

in the hUb address of the <:luster at the 

remote end of its hW ^ ^^^^ the high speed linlc. 

build. ««! transmits^ used by a FMC in the hub to perform 

'^^'r^.^^ZZTZtioL is established durihg 
the routing ^'»'*^*»"J"^!„J^ address cAecIcing can also be 
configuration programing. situation where hubs are 

, disabled in a «^ ^J^T^Ls of a KMC i. a hub 

S r::t'cor:ra;- arignored by the ... no 

s "^-nni^i::::.^-::^-----^^^^^^^^ 

" ■ ' ^ 4.^^ i*r<5-TT fflultidrop console linx. tme ovne* 

15 cMMwl o» , . . mtmiav to •llOW OOMOl. 

„ 11*.) r^«^'-^'''''^'^'^Ji:^'^r2^. 
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Uric.) Tne Ric «eua.^» ~ * .„^..t*rs-ll Clusters. In each 

^ ^^^^^ to flow between nodes In separate MCS-II ciusi^et 
traffic ^l^^'''^^ ^ ,,3.age. intended for remote nodes and 

transmits them as pacjw^» ^ ^.w™ .nd send the messages 



,1 to th. a..ti«tl«. n«J«. „ ,£ . POHCS 

uitut. -,?"^"f2i'^; „.t.r. ™= 



Initiate ...wg. „,t.r. TR. rac 

,1 serial linJc; when the FMC recei buffers 
« that a Cluster PHC's slave address on the multidrop 
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1 bus node Id. 

2 The handling of multidrop message data in the FOttCS network 

3 is basically the saine as that for general purpose async data and 

4 interrupts i Multidrop data is routed through the FOMCS network in 

5 st>ecial pac)cet5* Each multidrop data packet contains a destination 

6 clusteir address (which is validated by th6 destination FMC) and 

7 part of a message* tJhllke interrupt packets or packets of gtaeral 

8 purpose async data^ multidrop message packets also contain a source 
i cluster address. The receiving FHC uses the source address to 

10 segregate pieces of messages from different clusters into different 

11 buffiers. 

12 The sbiu^ce* cluster address Which a node inserts into a 

13 message is supplied by the FMC in the cluster. The multidrop 

14 master periodically sends a request-^cluster-address message to the 

15 FMC which causes it to broadcast a response message containing the 

16 cluster address to all nodes in the cluster. This cluster, address 

17 is the atddress eistablished during configuration programming via the 

18 - Define Cluster Address command. Hote that in a MCS-*II cluster 

19 where there is no FMC to supply the source cluste. 

2D communication between nodes is unaffected because the source 

21 cluster address field is hot included in messages exchanged by 

22 nodes in the same cluster. 

23 In FOMCS configurations where a hub is present, multidrop 

24 data is routed from the receiving FHC to ^e other FMCs in the hub 

25 via the hub MC bus. Packets are then built and transmitted from 

26 FMCs in the hub to the FMCs in the destination clusters. As with 

27 general purpose async and interrupts, a FMC in the hub looks at the 

28 detstination cluster address to determine if it should build and 

29 transmit a packet over its high speed link. If the destination 

30 cluster matches the address of the cluster connected to the FMCs 

31 serial data link, the FMC forwards the multidrop data. 

32 The MCS-II multidrop broadcast capability is also supported 

33 by FOMCS. When the FMC in a cluster receives a message over the 

34 local multidrop link that is to be broadcast to all nodes, it sends 

35 the message across its high speed link in packets which have a 
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^ *, ^ F«C8 in a hub will always forwatd such pacJcets 

broadcast flag sat. ^ ^^^inat ion duster field. All 

I^Wl^ation progra«dng. a cluster FHC is prograi»ed 
' a^oTl^e Cluster addresses. When Multidrop support 
\^!r ZJZc intercepts any message on the multidrop linlc 
is enabled, the FMC inT:er p ^i„^rn iii the list. 



U enabled, ^^r^^^^Hode in^e the cluster* iii the list. 
Which i'^*^^^^;;^,^^^ 2lt the Ust fr«M the PHC by 
: r-LSVr^^V F^C responds by encoding the 

S l^^tt a^^uster-list message which it .ends over the 
" »ultidrop li^. ^ ^^^^^^ configuration whic* allows 

.omaSc f^toT to a secondary high speed serial lin3c should the 

14 »»t«>»*^^*' f t T«« this high availability feature U 

15 primary link go down. This nig 

xe ^ll'^^*^"* Cluster monitor the heal^ 

1^ The seconda^ ™^J^** configured for the same MC bus 

,3 P'^f-^. «"*th.TriX i^^^ secondaries do not 

Ai-Hftr tvoes Of packet traffic (i.e., ^ 

=1/= .eal. the 

p.i^L^i:nriy s^d --;-^-;rr 

-J--- -/JtrtrJli^. - r^thLec. lin.. 
r^r:. -eo^pL'drcally exchange test pa^ets to determine 

if the primary linlc is still ^--^^^^'^^^ „^.^ i^^cation 
« a secondary FMC does not ^^^^^^^^^ '^^/^^^ ^r if the 

Within a specified -^^^ ::d^^^^^^^ P-«-' - 

secondary receives a 1^ ^^^^ , X> farcing the 

secondary initiates the fail ov^ P j^ix«,e-detected 
primary FMC to cease operation, and 2) senaing 
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1 packet to the secondary in the remote cluster* The remote 

2 secondary FMC th^ forces its primary FIfC to cease operation* the 
•3 remote secondary thein sends a fail^over-^complete packet back to 

4 the secondary which detected the failure* Subsequent communication 

5 between the clusters occurs over the secondary high speed link^ 

6 A secondary FMC forces a primary to cease operation by 

7 - assailing a special additional signal which is included in the 

8 health-check link cable conne^cting the primary and secondary.. When 

9 asserted this signal places the primary PMC into a 
-10 hardware-controlled offline state* In this state, all I/O 

11 interfaces on the FMC are disabled so that the FMC is unable to 

12 assert any signals oh the MC bus, transmit over the high speed 

13 link, transmit over any of its asyiic links or pulse any of its 
. ;14 output interrupt lines. The effect is identical to that achieved 

15 by putting the FMC online/of fline switch into the offline position. 

16 The (old) primary FMC is held in this offline state until the 

17 secondary FMC (which is the new primary) is reset via a power 

18 cycle, hardware risset or a received Reset command. 

19 While the old primaty FMC is being held in this offline state, 

20 it monitors the health-'Check link (if it is operational enough to 

21 do so) for the receipt of okay** characters from the new 

22 primary. If the old primary receives such characters, it 

23 reconfigures itself to assume the secondary role. Thus, when an 

24 attempt is made to return the old primary FMC to an online state, 

25 it behaves as a secondary and remains in a (firmware-controlled) 

26 offline state* This prevents the old primary from contending with 

27 ..the new primary on the MC bus* 

28 The faii-ovte process can also be initiated manually via 

29 the configuration programming link^. To cause a fail-over, a 

30 command is sent over the programming link to one of the secondary 

31 FHCs. The fail-over proceeds as described above and when it is 

32 complete, a response indicating such is returned via the 

33 programming link. 

34 While the previous discussion relates to high availability 

35 of the high speed link between clusters, the high availability 
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1 feature can be ^tended to the hub in a FOKCS star networX. High 
\ availability in a h^ is achieved by having a secondary n« f« 
\ each primary JHC in the hub. Thus, each cluster is connected to 

4 Sfhub b^tw*^ high. *eed lihte, a priory and a s^conda^;. 

5 ^Il-over to a secondary link is essentially the sa>e as the 
i; cluster to cluster case, 

■ Hbte that to really achieve high availabiUty xh a cluster, 

8 nodes Which have async or interrupt connections tc the prUary 
I ^MC must have a redundant set of connecti«» t* the secondary. 
,« ^ tha secondary PHC takes over, the nodes must switch over to 
r^^atn^ry ^c and interrupt connections. Xof^iUtatj^^ 

« procass, one of the output interrupt lines of a secondary ^ can 
i b^usad to inform the node(s, that faU^ ha. occurr^. 

14 configuration ptogra».dng cf a secondary nic, an output int«nn*t 

15 line can ba designated for this purpose. 

« aae faU-over process is not automatically reversible, 

once the sacondary FKCs have talcen over, they becomi in '"^^ 
primarias. When the ptavious primary WCs an«Vor high speed link 
are rapaired, they must be programmed to behave as the secondaries 
(if thay have no already reconfigured themselves as secondaries). 
21 configuration of a FMC as a primary or ^ 

« during configuration programming. The maximum time period that a 
!! »llow between -!•» okay- iiidications from the 



17 
18 
19 
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secondary will allow between «*ay- indications ^ 

24 ^^^is also estabUshed. the high availabUity featute can 

I -"^ ^^rin^ -'microprocessor is to allow diagnosis 
of an ^^tidual'mc and/or high spe^ sarial ^^ ^^^^ 
bringing the entire MC8 network offlina. This capability is 
pa^Slarly useful in the star network environment where it ^ 
^lasCable to shut down the entire hub just to diagnose a problem 

31 Land a particular cluster. « prObl« t^ out 

32 to be the high speed link or the iHC in the duster, it can be 
corrected without taking down the entire hob. Of course, software 
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35 



corrected wimoro x.a«n^ .#*o^4-of^ 

r..,n=h»»l»tion will b. ..c«i.«y b."-" ^' '"r^ 

ol,««r th. oth« olort«. it «• c«««mieatl,w with Wt tk. 
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1 teat of the clusters connected to the hub can continue to 

2 cbsuiiUnicate without interruption. 

3 special operational nodes, referred to as hub diagnostic and 

4 cluster diagnostic modes, are available to allow the FMC to b^ 

5 isolated from the high speed coax or fiber link for testing 

6 purposes. In these modes, the FMC operation is identical to normal 

7 hub or clusteir mode except that th6 output of the Gazelle Hot Rod 

8 Tx chip lis directly Connected to the input of the Rx chip. To 

9 configure the FMC to operate in hub. of cluster diagnostic mode, the 

10 Set FHC Online/Offline command used to place the FMC in a 

11 firmware-controlled offline state. Then, the desired mode is 

12 selected via the Set Operational Mode command and the FMC is 
.13 returned to an bhiine state by m^ans of a second Set FMC 
:T4 Onilne/Offiihe command.. 

15 Individual functional areas of the FMC can be tested by 

16 sending Specify Diagnostic Loopback Data commands to thB FMC. 

17 Variations of the Specify Diagnostic Loopback Data command can be 

18 used to invoke memory write traiisfer loopback, async data loopback 

19 and intisrrupt 16opback. A FMC will only accept such, commands when 

20 • in the firmware-controlled offline state. 

21 the FMC supports three diagnostic loopback modes for memory 

22 write transfers: internal, external and MC bus loopback. Internal 

23 loopback loops data through the FMC from the KC bus input latch to 

24 the MC bus output latch. External loopback tests the same path 

25 excejpt that the data is actually transmitted over the liigh speed 

26 serial link and looped back via an external loopback cable 

27 connection, internal loopback mode is shown in Figure 22. 

28 External loopback niode is shown In Figure 23. MC bus lobpback 

29 loops data from the MC bus output latch to the MC bus input latch 

30 via the MC bus. This loopback mode is illustrated in Figure 24. 

31 Note that when internal or external loopback is performed, 

32 the FMC hit/translation RAM must also be programx^ed appropriately 

33 to achieve the desired effect. Loopback modes can be used to 

34 specifically test reflection region or address translation logic 

35 or the hit/translation RAM can be programmed such that all regions 
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i. ai.2l-^L 0.T.11. not »a Chip. .r. co„f l,»r.a .«ch 
. «« th. tr»»itt« Chip i. 

! Sat tM r«.iv.r chip. tt. S*«l« Dl^nctlc 

!r^^^«t.pitt«.t.h.l««»athr««hth.iMCh.rd»r. 
y!!Jr Th. .aar... "a a.ta proc«a thiou*. th, « p.th «aj^t 

ftc. th. i.t,* »a i«=iua«i th- in «« .«.t b.c» 

» "^jr^^xc^Si- "= *^ ™i 

17 in extern ^ , ^ .t^e Loppback Data coimand 

" :LrSS;t.^^'Cr^t.lr„ aaa^.. «.a a.t, p.tt.» 
f^t ^^^hro«h th. FKC h«a.«. «a th. «t«n.i locpw* 

! ^rtS i«.«t. th. .aar.- «a a.t. p.tt«* i«tc th. 

41 co»n.ctl«.. tt. ™= 1 ,j„^ ^ » 

" r^«'rxr>lh- X «.pc«. u «« h«» VI. th. 

" Tr.^^ th. aaar^ a.t. P.tt«n 

« h». lccpb.ck .ca., th. hi*. W«a .«:l.l llnlc l«t«*.c. 

" ^-iBrLrrriTtTTh.'^t- 

35 When it receives a grant, generai^e^ «^ 
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1 thd data valid signal deasserted. This causes any nodes on the bus 

2 to ignore the transfer. The FHC MC bus interface hardware, 

3 hovever, is configured to receive its own transfer. The looped 

4 data is read from the KC bus input latch and retttmed in the 

5 response sent back via the prograauning link. 

6 The FHC supports three diagnostic loopback modes for general 

7 purpose async data; internal # external^ and port-port. The three 
a varieties ere illustrated in Figure 25. The FHC performs the 
9 requested async loopback as the result of a Specify Diagnostic 

10 Xioopback Data command rweived over the conf iguration programming 

11 link. 

12 thet Sjpecify Diagnostic Lop^>ack Data commamd informs the FMC 

13 as tb %^ich async port(8) will participate in the loopback test and 

14 provides the data to be looped. If an async port is selected for 
is internal loopback, the FMC initializes the port hardware to wrap 

16 tranismitted data back to the receive side of the port. The FMC 

17 then causeB the port to transmit the data. if the port is 

18 functiohihg correctly, the FMC immediately receives the data back 

19 again. The received data is passed back over the programming link 

20 in the command response* If no data is received, the FMC returns 

21 a response indicating such. 

22 External loopback requires that a loopback plug be connected 

23 to the port lAiich wraps transmitted data baek to the receive pin 

24 of the port. The FMC initializes the port hardware fot normal 

25 operation and transmits the data via the specified port. Again, 

26 the data should be immediately received from the same port. The 

27 response to the command contains the received data or indicates 

28 failure of the loopback operation. 

29 Port^port loopback requires that a RS^232 cable connect the 

30 two selected ports. The FMC initializes the ports for normal 

31 operation and transmits the data via the port specified in the 

32 Specify Diagnostic Loopback Data command. The responsie to the 

33 conmand contains the data received from the other port or indicates 

34 failure of the loopback operation. 

35 The FMC supports external loopback for the interrupt lines. 



Wd 93/19422 



PCrAIS93/02839 



32 



1 



K specify Diagnostic Loopback Data coMoand to the FMC selects a 
2 p^irTt iSerrupt lines that will participate in the loopbaek, one 
'i^^ut a,^ one !«tput. Hote that loophac3c require, ^that e «i^^ 

4 ^ect the selected i«terr,«>t lines. To perform the loopbacK 

5 ^ JHC geherates a pulse on the output line of the ^Ix and 
e repc^rts in its co«»a«d response whether or not a pulse was detected 

7 on diagnostic loopl«clc «od«i f or the HC8-II 

I wittarop console link: internal and ertemal. Internal loopback 
is p^i^br-ed exactly as with the general purpose async ports. 
BctLal loopback is functionally identical to the .ethod used with 
the general purpose async ports, but because of the «nx«ue nature 
of the multidrop link, no external loopback ««^^* ""^^V 
H6wever, the ertemal loopback test will drive the multidrop link, 
therefore, if testing is desired without bringing the entire 

lii* Offline, the MCS cable which carries the nUltidrop 
link Should be unplugged from the mc chassis l,ackpl«»e prior to 
initiating diagnostic test. «.d replaced when testtog is 
„ The WC support, two diagnostic loopback mcKies for «.e high 

20 availability health-a»eck link; internal and external. Both modes 

21 ^"etf ormed in the s«»e manner as with the general purpose async 

22 IPortB, across the high speed serial link in 
a FfiHCS coJtUation contains an error detection code (BDC) which 
Llows the receiving FMC to determine if an error ^-^^^^ 
transmission. Packets received with good EDC arc also 
S^al or iUogical bit settings. P«:ity is c*ec*ed for^nsf ers 
liSvid the MC bu. and transfers received with good parity 

^, are also *h«*«i for Ulegal or ilicgical bit 
30 Por packet errors and transfer errors detected by 

\[ receiving 4, the FOHCS philosophy is to report «.e ^-J^ ^ 

32 noda(8) in the nearest cluster rather than reporting the error to 

33 nt i^efin the network, ho attempt U made to handle errors in 
:l^«it.tr««fer packets differently from errors <l^^^ 

35 oth«: types of packets b««iuse once an error is detected in a 
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1 packet, the entire contents of the packet are suspect including the 

2 packet type field. 

3 The method of reporting errors is illustrated in Figures 26 

4 and 27. When a FHC in a cluster detects an error in a packet 

5 received over the high speed link or detects an error in a MC bus 

6 transfer, it reports the error to the node^s) in the cluster by: 

7 1) forcing a parity error on the MC bus and/or 2) directly 

8 interrupting the hode{s). To force a parity error, the FMC 

9 arbitrates, for the MC bus and vhen granted access, generates a 

10 transfer vhere the parity driven on the bus does not natch the 

11 address and data* miis causes the MC port of the each node to 

12 detect a transfer with bad parity and (hopefully) report it to the 

13 node via a parity error interrupt. The direct interrupt approach 
,14 utilizes one or »ore of the FMC's eight output interrupt lines. 

15 iQien a PMC in a hub detects an error in a transfer received 

16 over the hub MC bus or detects an error in a packet receivjed over 

17 its high speed link, it builds an error packet and transmits it to 

18 the FMC at the other end of the high speed link. Receipt of the 

19 error packet causes the FMC in the cluster to report the error as 
20^ described earlier. Note that whether or not the FMC in the cluster 

21 receives the packet correctly, an error will be reported. 

22 To assist with diagnosis of packet transidission or MC bus 

23 transfer errors ,^ the FMC keeps a copy of the last bad packet (if 

24 any) and the last bad MC bus transfer (If any) that it received. 

25 These copies can be accessed at any time via the. configuration 

26 programming link. 

27 During configuration programming, the error reporting 

28 behavior of a FMC in a cluster is established. The FMC can be 

29 programmed to report errors by generating MC bus parity errors 

30 and/or by generating interrupts. If the interrupt approach is 

31 selected, one or more output interrupt lines of the FMC can be 

32 programmed as outputs for error signals. Each of the selected 

33 lines can be further qualified as to when the line is pulsed: l) 

34 when a bad MC bus transfer is detected (or when an error packet is 

35 received indicating that the remote FHC detected a bad transfer) , 
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w ^ -.v^i^ tfi received over the high speed link (ot when 

a .:r::^ror 3, elt^ a bad transfer or a bad pacXat i. 

5 "^'"^^^gh th* present inver^tion has be.n shown and described 
with Jifot Jce to preferred embodiments, c^ahges and modifications 
lift^^le to those smied in the art whic* do not dspart ^ 
Se W*^ and C0nte»platio*» of th. i«v«»tiv Concepts taught 
^eih such are de«>»d to fall within the purview of the 
invention as claimed. 
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We claim: 

1* A systexQ coroptlsing: 

a first and secoild set of plurality of nodes; 

a first data bus associated vith and connecting said first s^t 
of plurality of nodes; 

a second data bus associated with and connecting said siecond 
set 6f plurality of nodes; 

each node of said sets of plurality of nodes including a 
processing unit, a memory, a bus coupled to the processing unit and 
memory, and a sensor means for sensing a write to the memory and 
for transiaittixlg the sensed write on said associated data bus; 

first converter means connected to the first dat& bus for 
converting data On said first data bus to corresponding optical 
signals and received optical signals to corresponding data for 
transmission On said first data bus; 

second converter means connected to the second data bus for 
converting data on said second data bus to corresponding optical 
Signals and received optical signals to corresponding data for 
transmission oh said second data bus; and 

fiber optic means for optically transmitting data from one 
converter means to the other converter means. 

2. The systeib of claim 1 wherein each said converter means 
includes conversion means for converting parallel data to serial 
optical signals and vice versa. 

3. The system of claim 1 wherein each said converter means 
includes a FIFO for temporarily storing data. 

4. The system of claim 1 wherein each node further includes I/O 
means for introducing I/O data- into memory and wherein said sensor 
means responsive to a write to memory of I/O data transmits same on 
said associated data bus. 

5. A system for connecting memory coupled systems, comprising; 
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a plurality of nodes; 



iibdes; 



a f i^st data bus connecting a first gtoup of said plurality of 
a 86c&nd data bo. connecting a second group of said plurality 

first converter means connected to the. first data bus, 

second converter means connected to the second data bus; 

first fiber optic means for carrying transmitting data from 
th& first converter means to the second converter means; 

second fiber optic means for carrying transmitted data from 
the second converter means to the first convert^: means, 

the first and second converter means each comprising: 

input latch means for receiving data- from a respective 

data bus. ^ translation RAM means connected to the input 
latch means for determining the destination of data received from 

taie data bus; , ^ , ^ w 

first micro-interface means connected to the input latch 

fisan^ for controlling the hit and translation RAK means; 

transmission FIFO means connected to the input latcSh 
«eans for latching the data received from the data bus; 

error detection means for determining if an error exists 
in the data in the transmission FIFO means; 

first and second transmission latches connected to the 

transmission FIFO means; - 

transmitter means connected to the first and second 
transmission lat<Aes for transmitting the data to another converter 

receiver means for receiving data transmitted from 

ahothfer converter means; i-he 
first and second receiver latcihes for latching the 

error detection means connected to the first and second 
latch means for checking if an error exists in the received data; 

second micro intetf ace means far testing the received 
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data if the bheck by the error detection means fails; 

receive FIFO means connected to the first and second 
receive latch mean9 for holding the received data after checking by 
the errot detection means; 

output latch means for" transmitting the received data to 
the respective data bus. 

6. A system as claimed in Claim 5, further comprising first and 
fifecbnd bacJcup controllers for transmitting data between memory 
coupled systems upon a determination the first and second 
controllers are not working properly. 

. 7. A system as claimed in Claim 6, wherein said input, output, 
tirainsmit arid receive latches of each of said plurality of 
cohtrdllers are able to be accessed in both a parallel and serial 
fashion. 

8. A system a^ claimed in claim 1, wherein data is transmitted 
through the optical fiber means in 80 bit data frames. 

9. A method co&prising the steps of: 
establishing first and second data links; 

establishing first and second sets of nodes, each node 
ihbluding a processing unit, a memory, a bus coupled to the 
processing uiiit and memory and a sensing means for sensing a write 
to memory; 

sensing a write to a memory in one of said nodes of said first 
set; • 

transmitting said sensed write on said first data link; and 
sensing the sensed write transmitted on said first data link 
and optically transmitting same to a remote point whereupon it is 
transmitted on said second data link to the memory of one of the 
nodes of said second set without intervention of the processing 
unit of said one node of said second set. 
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10. The Bethod Of cXaiB 9 comprising the further steps of: 

Lting I/O d.ta fro» an I/O souroe into the ^eiaory of a node 

'1.^irth"t^teTi%'aata and tran^itting sa»e on the data 
link associated with said node; and 

of the oOir <M. of noi- via tt. .«oci.t.d Ut» 
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